feat(server): build, sign, and submit Bulletin preimages in the core#270
feat(server): build, sign, and submit Bulletin preimages in the core#270pgherveou wants to merge 150 commits into
Conversation
Adds the canonical testing module (api/testing.rs) and its v01/v02/versioned wiring used by the Rust host runtime and generated clients.
New crate defining the host syscall traits (storage, navigation, consent, permissions, ...) that host runtimes implement. Types are re-exported from truapi::versioned/v01 rather than redefined.
WASM host runtime that hosts implement: dispatcher, SCALE frames, subscription streams, chain runtime, host logic (sessions, SSO pairing, permissions, statement store, dotns) and the wasm bindings. Includes the committed generated dispatcher/wire-table under src/generated/.
…backs Extends the rustdoc-JSON code generator to emit the Rust dispatcher and wire table consumed by truapi-server, plus the TS host-callbacks adapter. Golden tests pin the emitted shapes.
New WASM-backed host runtime package embedding the Rust core, with web iframe and Web Worker entry points. Updates the @parity/truapi client (SCALE, sandbox, transport) and drops the obsolete explorer 0.3.2 codegen snapshot.
Updates CLAUDE.md/README, CI workflows, Makefile, deny.toml, changesets, and linguist attributes for generated code, and bumps the dotli submodule to the host integration that consumes the WASM runtime.
Adds the canonical testing module (api/testing.rs) and its v01/v02/versioned wiring used by the Rust host runtime and generated clients.
New crate defining the host syscall traits (storage, navigation, consent, permissions, ...) that host runtimes implement. Types are re-exported from truapi::versioned/v01 rather than redefined.
…backs Extends the rustdoc-JSON code generator to emit the Rust dispatcher and wire table consumed by truapi-server, plus the TS host-callbacks adapter. Golden tests pin the emitted shapes.
Move the entire Bulletin TransactionStorage.store submission into
truapi-server. The core builds the extrinsic offline (subxt 0.50.2), signs
it with the wallet-delegated allowance key, dry-runs it, broadcasts, and
watches for inclusion over the existing chainHead runtime — replacing the
host signer-callback seam so the allowance secret never crosses the
host/FFI boundary.
Core:
- host_logic/extrinsic.rs: offline SubstrateConfig assembler with a
config-pinned genesis hash, an sr25519 Signer, and metadata / transaction
validity / events / header decoders.
- host_logic/bulletin.rs: store{data} construction signed with the allowance
key, with audited pallet/call-index pinning plus a canonical-bytes guard so
provider metadata cannot redirect the signature, and a memcpy call-data
encoder that avoids scale-encode's per-byte cost.
- runtime/bulletin_rpc.rs: serialized submit flow (ephemeral with_runtime
follow, metadata, nonce, validate_transaction dry-run, broadcast, single
event-loop inclusion watch gated on nonce advance, System.Events dispatch
check), typed error taxonomy, and broadcast stop on every exit.
- runtime.rs: Preimage::submit gates on bulletin availability before any
prompt and refreshes the allowance (Increase policy) with one retry on an
allowance rejection; lookup_subscribe verifies blake2_256(value)==key and
serves an in-core content-addressed cache.
- BulletinAllowanceKey is zeroized on drop; PreimageHost keeps only
lookup_preimage; both host configs gain an optional Bulletin genesis hash.
Codegen/TS: regenerate goldens (product wire unchanged) and drop the signer
bridge from the handwritten host worker. CI compiles the crate for
wasm32-unknown-unknown; .gitignore ignores the renamed wasm bundle path.
Bumps the hosts/dotli gitlink to the matching submodule commit.
Adversarial review of the watch loop found two defects: - A crafted or buggy chain provider could send a self-referential or cyclic `NewBlock` parent link. The nonce-advance ancestor walk had no visited-set guard and no `.await`, so such a link spun forever, freezing the worker and permanently holding the submit lock across all products. The walk is extracted into `ancestors_to_check`, which guards against self-parent and cyclic links with a visited set, and is unit-tested. - Blocks that failed the nonce gate were marked checked but never unpinned, leaking chainHead pins over the watch's lifetime and risking a false BroadcastUnverified once the server's pin limit was hit. They are now unpinned like body-negative blocks. Also log a warning when a host lookup value is downgraded to a miss for failing the blake2_256(value)==key integrity check.
…bler The signing-host role now builds and signs transactions locally instead of returning Unavailable, reusing host_logic/extrinsic.rs. Because ProductAccountTxPayload carries each extension's `extra` and `additional_signed` already SCALE-encoded in canonical order, assembly is a pure offline concatenation — no metadata, no RPC: - extrinsic.rs gains build_signed_extrinsic_v4 + v4_signer_payload and an Sr25519Signer::from_keypair constructor. Body = Compact(len) ++ 0x84 ++ MultiAddress::Id(signer) ++ MultiSignature::Sr25519(sig) ++ Σextra ++ call_data; signer payload = call_data ++ Σextra ++ Σadditional_signed, blake2_256 only when >256 bytes. Layout is byte-identical to subxt / frame-decode. - signing_host::create_transaction handles Product and LegacyAccount (with a fail-closed slot-zero key-match check); the product-facing entrypoint's caller-scoping, chain-submit permission, and user-confirmation gates already precede it. - Extrinsic V5 (tx_ext_version != 0) returns the new AuthorityError::NotSupported -> HostCreateTransactionError::NotSupported: V5 general carries the signature inside a VerifySignature extension, which cannot come from pre-encoded parts. Tested: v4 layout + signature verification, the >256 hashing boundary, extension order preservation, and Product/LegacyAccount success plus v5/mismatch/no-session rejections.
0836131 to
60b43ea
Compare
…core # Conflicts: # .github/workflows/ci.yml # .github/workflows/release.yml # .gitignore # CLAUDE.md # Makefile # hosts/dotli # js/packages/truapi-host/scripts/build-wasm.mjs # js/packages/truapi-host/src/adapter-support.ts # js/packages/truapi-host/src/host-callbacks-adapter.test.ts # js/packages/truapi-host/src/runtime.ts # js/packages/truapi-host/src/web/create-iframe-host.test.ts # js/packages/truapi-host/src/web/create-iframe-host.ts # js/packages/truapi-host/src/web/create-worker-host-runtime.ts # js/packages/truapi-host/src/web/worker-provider.test.ts # js/packages/truapi-host/src/worker-protocol.ts # js/packages/truapi-host/src/worker-runtime.ts # js/packages/truapi/src/sandbox.ts # playground/src/lib/auto-test.ts # playground/tests/e2e/dotli-diagnosis.ts # playground/tests/e2e/dotli/helpers/signer-bot.ts # rust/crates/truapi-codegen/src/rust/wasm_bridge.rs # rust/crates/truapi-codegen/src/ts/host_callbacks.rs # rust/crates/truapi-codegen/tests/golden/host-callbacks-adapter.ts # rust/crates/truapi-codegen/tests/golden/host-callbacks.ts # rust/crates/truapi-codegen/tests/golden/wasm_bridge.rs # rust/crates/truapi-codegen/tests/golden/worker-callbacks.ts # rust/crates/truapi-codegen/tests/golden_rust_emit.rs # rust/crates/truapi-server/src/runtime.rs # rust/crates/truapi-server/src/wasm.rs # rust/crates/truapi-server/src/wasm/generated_bridge.rs
Keep physical provider ownership in hosts while moving shared chain behavior and Subxt-backed handling into Rust. Harden Bulletin retries and host lifecycle coverage.
| let elapsed = start.elapsed(); | ||
| assert!(signed.encoded().len() > data.len()); | ||
| assert!( | ||
| elapsed < std::time::Duration::from_secs(5), |
There was a problem hiding this comment.
I think this is why the CI is failing. Maybe consider bumping as it currently taking 6.7 sec
| elapsed < std::time::Duration::from_secs(5), | |
| elapsed < std::time::Duration::from_secs(10), |
valentinfernandez1
left a comment
There was a problem hiding this comment.
Overall looks good to me just some small nits.
| /// build + dry-run + broadcast + best-block inclusion. Passed explicitly to the | ||
| /// chain layer, which uses the call context only for cancellation. | ||
| const PREIMAGE_SUBMIT_BUDGET: Duration = Duration::from_secs(180); | ||
| /// Preimage submission is exercised by host diagnosis with a 240s live-chain |
There was a problem hiding this comment.
This comment seems more related to the truapi-playground diagnosis, not sure it makes much sense here.
| people: { | ||
| genesisHash: string | Uint8Array; | ||
| }; | ||
| /** Bulletin-chain genesis hash used for in-core preimage submission. */ |
There was a problem hiding this comment.
Not a blocker but let's make sure to mark this as a breaking change. Also can we add docs to the rest of the fields here? 🙏🏻
| /// | ||
| /// // Submit a preimage first so the lookup resolves to a value. | ||
| /// const submitted = await truapi.preimage.submit("0xdeadbeef"); | ||
| /// const value = crypto.getRandomValues(new Uint8Array(4)).toHex() as `0x${string}`; |
There was a problem hiding this comment.
I think the previous example was better
| /// | ||
| /// ```ts | ||
| /// const result = await truapi.preimage.submit("0xdeadbeef"); | ||
| /// const value = crypto.getRandomValues(new Uint8Array(4)).toHex() as `0x${string}`; |
| [target.'cfg(target_arch = "wasm32")'.dependencies] | ||
| futures-timer = { version = "3", features = ["wasm-bindgen"] } | ||
| js-sys = "0.3" | ||
| subxt = { version = "0.50.2", default-features = false, features = ["web"] } |
There was a problem hiding this comment.
We should measure the impact of this on the bundle size as I am guessing it will increase it significantly, not an issue on desktop and mobile but it might on the browser
What
Preimage submission to the Bulletin chain now happens entirely inside the Rust core (
truapi-server).The core builds, signs, and submits the
TransactionStorage.storeextrinsic itself, routing chain traffic through the host's existingchain.connectJSON-RPC pipe.Previously the core handed the host a signing capability and the host built and submitted the transaction with PAPI.
That seam is gone. As a result:
Ownership: before vs after
The topology is unchanged (the core still reaches the network only through the host).
What changes is ownership.
Before, the secret bytes were core-owned but the capability to use them, and the decision of what gets signed, was host-owned:
After, everything signing-related sits on the core side; during submission the host just forwards opaque JSON-RPC text:
Submit flow
Changed surface
host_logic/extrinsic.rs: offline subxt assembler (config-pinnedSubstrateConfig, sr25519 signer, metadata / validity / events / header decoders).host_logic/bulletin.rs:store{data}build + sign; the call is resolved by name from the fetched metadata, with a hard check that thedataargument really is a byte sequence.runtime/bulletin_rpc.rs: the submit flow above + typed errors.runtime.rs:Preimage::submitordering with one allowance refresh + retry on rejection;lookup_subscribecache + integrity check.truapi-platform:PreimageHostkeeps onlylookupPreimage;BulletinAllowanceKeyzeroized on drop; host configs require the Bulletin genesis hash (runtimeConfig.bulletin.genesisHashon the TS side).@parity/truapi-host) + dotli submodule: signer bridge removed; dotli keeps content lookup only (smoldotbitswap_v1_get, or IPFS gateway in RPC-gateway mode) and serves Bulletinchain.connecton both its light-client and RPC-gateway backends.